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(57) ABSTRACT 

In a frame-switched network, a sender sends frames to a 
receiver over a possibly unreliable channel. Sent frames 
include frame identifiers that can be used for a limited 
automatic repeat request. Upon receipt of a frame, the 
receiver determines, from the frame identifier, if frames 
prior to the received frame were lost in transit. If the receiver 
determines that it missed a prior frame, the receiver sends 
the sender a negative acknowledgment (nack) for the missed 
prior frame or frames. Otherwise, if the receiver receives a 
frame correctly, it does not acknowledge the frame. The 
frame identifiers can be a set of sequential integers with 
frames transmitted in sequential frame order. In some 
embodiments, when a receiver receives a frame out of order, 
the receiver buffers the out of order frame in a receiver buffer 
for a receive buffer period until preceding frames are 
received or a receive buffer period expires. The sender can 
send a reminder frame to the receiver to allow the receiver 
to detect a missed prior frame missing from an end of a 
frame sequence. The channel between the sender and the 
receiver can be a bidirectional channel over a telephone 
wire, a cable, a radio frequency link or a power wire. 
Multiple logical channels might be set up between a given 
sender-receiver pair, to allow for traffic of varying priorities. 
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FIG. 3 
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LIMITED AUTOMATIC REPEAT REQUEST 
PROTOCOL FOR FRAME-BASED 
COMMUNICATION CHANNELS 

BACKGROUND OF THE INVENTION s 

The present invention relates to communications systems 
in general and, more specifically, to methods and apparatus 
for reducing data loss on a network with an unreliable 
physical layer. 

Many frame-handling schemes and protocols have been 10 
used for data transport. In systems implementing those 
schemes and protocols, data is "packaged" into frames and 
sent from a sender to a receiver (or to a plurality of receivers 
in a broadcast mode). When the receiver receives a frame, it 
unpackages the data and uses it. In the standardized ISO 
seven-layer network, a transmitting application moves data 
to a receiving application by providing the data to a layer 
below an application layer, which in turn provides it to the 
next lower layer, until the data reaches the physical layer and 2Q 
is sent to the receiver. At the receiver, the process is 
reversed, with the data flowing up from the receiver's 
physical layer to its application layer. The seven layers, from 
the bottom to top are: 1) the physical layer, 2) the link layer, 
3) the network layer, 4) the transport layer, 5) the session 25 
layer, 6) the presentation layer, and 7) the application layer. 
FIG. 1 illustrates this concept. 

Much has been written on the benefits of such an arrange- 
ment of layers and need not be recounted here. Many 
standards for data communications between peer layers have 30 
also been proposed and are in common, use. The most 
common form of networked communication in a local area 
network in use today is Ethernet/IEEE-802.3 network pro- 
tocol communication ("Ethernet" for short). Because of its 
ubiquity, many Ethernet components are readily available at 35 
low cost. For example, single chips are readily available that 
perform much of the logic needed to implement Ethernet 
communications. Furthermore, because of their ubiquity, 
those chips have been widely tested and the technology is 
mature to the point where its usefulness and limitations are 40 
well known, in addition to being readily available. 

The Ethernet link layer was designed to be supported by 
a relatively reliable physical layer below the link layer. 
Because of this, the Ethernet link layer, in its standard 
operating mode, does not correct errors in frames nor does 45 
it retransmit lost frames. A number of reliable physical 
layers have been standardized for Ethernet, including ones 
based on coaxial cables, high quality twisted pair, and 
optical fibers. These generally deliver very low error rates 
(e.g. less than 1 error ed frame per million frames sent). 50 
Consequently, many higher layer protocols have been 
designed with low error rates in mind and perform poorly at 
higher rates. For example, streaming video applications 
require low error rates, as the loss of 1 frame or more in 
1,000 frames can cause visible errors. As another example, 55 
the Transport Control Protocol (TCP), which is used for 
virtually all wide area (Internet) reliable transport, performs 
poorly when more than 1 frame in 20 is lost. If an unreliable 
physical layer were used with standard Ethernet link layer, 
higher layer protocols would have more lost frames. go 

With the growth of home networking, more and more 
computer users are expecting to be able to connect their 
home computers and other home appliances together. With 
the ready availability of inexpensive Ethernet components, 
the computer users might like to set up an Ethernet network 65 
for the interconnected home devices. However, unless the 
house is wired with standardized media such as CAT 5 
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twisted pair cables or the like, the network must be set up 
over existing wiring, which usually means either phone lines 
or power lines (unless the network is wireless). While phone 
lines and power lines can easily carry lower bit rate data 
streams, transmitting data at Ethernet rates is challenging, 
since neither phone lines nor power lines were designed to 
carry high bandwidth communications. The typical phone 
line is a set of twisted pairs terminated in RJ-11 sockets with 
no matching termination resistors and could be daisy- 
chained from phone jack to phone jack with many branches 
further degrading the lines and adding noise onto the lines. 
Power lines have their own set of problems with the power 
grid and appliances injecting noise onto the wires. 

The unreliability of these lines at high bandwidths results 
in frame loss or delay. Where the noise or other impairments 
on a line cause the receiver to make even a single error in 
decoding a frame's data, that frame is lost. Loss of frames 
results in loss of data or delay due to retransmission of the 
lost data. For time sensitive applications, delay may pose a 
significant problem. For other applications, such as UDP- 
based multicast of audio/video streams, frequent lost frames 
are a problem. 

Attempts to increase the reliability of an unreliable chan- 
nel generally fall into two categories: using error-correcting 
codes (forward error correction) or using a reliable data 
stream protocol. Error correcting codes require that extra 
data be sent, using up extra channel capacity for all traffic 
and extra computational resources in the sender even when 
there are no errors. Even with strong error correcting codes, 
frames may still be lost because the frame is too badly 
damaged or is simply lost because of errors at the beginning 
or end of the frame. Therefore, extra effort will be expended 
computing and transporting extra bits that will either not be 
used, in the case of undamaged frames, or cannot be used, 
in the case of badly damaged frames. 

Reliable data stream protocols, such as HDLC LAPB, 
provide a reliable data channel over an unreliable medium 
by having the receiver acknowledge receipt of frames and 
having the sender retransmit frames that are not acknowl- 
edged. A reliable data stream protocol is used to ensure that 
all frames are successfully received without regard to 
timeliness, and thus the protocol requires relatively frequent 
acknowledgments to be sent from the receiver to the sender, 
as well as flow control to halt the flow of data. These 
protocols also require explicit connection establishment 
between pairs of stations (a sender and a receiver) and do not 
support multicast applications. The above factors require 
extra bandwidth, may introduce delay and make the network 
less robust. For these reasons, the use of reliable data stream 
protocols has limitations as a method to increase reliability. 

Therefore, a protocol and supporting methods and appa- 
ratus are needed that can reduce effective frame loss rates 
and delays to the level of standard Ethernet frame loss rates 
and delays, over a physical layer that is expected to be 
unreliable, and while doing so at minimal cost in terms of 
network bandwidth and host resources, and while preserving 
the connectionless, low-latency, and multicast service nor- 
mally expected of Ethernet. 

SUMMARY OF THE INVENTION 

In a frame-switched network according to one embodi- 
ment of the present invention, a sender sends frames to a 
receiver over a possibly unreliable channel. A sent frame 
includes a frame identifier selected from a set of reusable 
frame identifiers. The sender stores the frame in a frame 
buffer for a buffer period. Upon receipt of the frame, the 
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receiver determines, from the frame identifier, if frames FIG. 10 is a flowchart of a LARQ frame transmission 

prior to the received frame were lost in transit. If the receiver process. 

determines that it missed a prior frame, the receiver sends FIG. 11 is a flowchart of a LARQ frame reception process, 

the sender a negative acknowledgment (nack) for the missed FIG. 12 is a frame flow diagram illustrating several 
prior frame or frames. If the sender receives a nack, the 5 examples of a frame handling process performed between a 

sender determines the frame identifier(s) of the missed prior sender and receiver 

frames) and resends the missed £rame(s) if the missed pjQ. 13 is a frame flow diagram illustrating several 

frames) is (arc) in the frame buffer. At the end of a buffer addilional exa ]es of a frame handli ^ perforin ed 

fr^°e 'buffer ^ 10 between the sender and receiver. 

T , i FIG. 14 is an illustration of one application of the frame- 

In some specific embodiments, the sender transmits the swi[ched netW£)rk rf ^ ( invention over a potentially 

transmitted frame to more than one receiver, such as in a „ r . i u ~i 

, . . , , ^ , . , ' , unreliable channel, 
multicast or broadcast mode, lne frame identifiers can be a 

set of sequential integers with frames transmitted in sequen- DESCRIPTION OF THE SPECIFIC 
tial frame order. In some embodiments, when a receiver 15 EMBODIMENTS 
receives a frame out of order the receiver buffers the out of Sevefal embodiments are describ ed herein for implement- 
order frame in a receiver buffer for a receive buffer period a q] feferred tQ herein && ^ . LARQ , m{ (for 
until preceding frames are received or the receive buffer Automatic Rcpeat rcQucst) M ^ bc appareilt 
period expires. upon reading this description, the LARQ protocol reduces 

In some embodiments, the sender sends a reminder frame the effcctive crror rate of an unreliable frame-based com- 

to the receiver to allow the receiver to detect a missed prior mun i cat ion channel or network. Unlike most sequence 

frame missing from an end of a frame sequence. The channel num ber-based protocols, LARQ does not attempt to guar- 

between the sender and the receiver can be a bidirectional antee reuaD i e delivery of every frame, but instead attempts 

channel over a telephone wire, a cable, a radio frequency to coyer up some losses b lhe physical layer through fast 
link or a power wire. Multiple logical channels might be set ™ retransmission of lost frames and leaves the reliable delivery 

up between a given sender-receiver pair, to allow for traffic of eyery frame l0 the higher network layers . This signifi- 

of varying priorities. cantly enhances the usability of networks that may, at least 

One advantage of the present invention is that new data occasionally, have loss rates of one in a hundred frames or 

can be transmitted between a sender and a receiver without worse, and provides this enhancement with minimal addi- 

the sender having to confirm that older data was sent and t j ona i complexity at the lower network layers, 

received correctly, while still providing a window of oppor- 0ne &uch application for the LARQ protocol is where 

tunity for frames to be quickly resent. The window of mtthods used for highly reliable channels are adapted for 

opportunity advances as new frames are sent. As a result of unreliable channels. For example, LARQ protocols can be 

the advancing window, some frames may be lost, but the UJJcd for implemerlting an Ethernet network (which usually 

protocol is designed such that those lost frames will be ^ on rdiable channels such ^ a coaxial cable ded i ca ted 

handled at a higher layer. No flow control is required at the tQ Ethemet ) ovcr te i C phone lines or power lines that are also 

sender end, therefore no channel setup or receiver flow used for analog telephone service or AC power distribution, 

control setup is required. RG 1 mustrales the 5asic stnicture of a aetwork stack 

A further understanding of the nature and advantages of 4o betwecn tw ti or commumC ations syst ems. In the 

the mvenuons herein may be realized by reference to the ISQ yiew Qf networking) an applica tion (AP "X") on one 

remaining portions of the specification and the attached syslem (System A) communicates application data to an 

drawings. application (AP "Y") on another system (System B). While 

BRIEF DESCRIPTION OF THE DRAWINGS the communication between the applications appears to the 

FIG. 1 is a diagram illustrating the standard seven-layer 45 applications to be peer-to-peer communication it is in fact 

network stack communication with the next lower network layer, in this 

FIG. 2 is a block diagram showing the standard data flow case u lhe P™""*** 1*5™. As data is passed down through 

in the lower three layers of a network stack. " ch ^ ° n ^ , A ' hat f f r encapsulates die data 

„ . , , , , « n j . Once the data reaches the lowest layer, the physical layer, it 

FIG. 3 is a block diagram showing a data flow according fa transmitted t0 B and tnen ihQ data works its 

to one embodiment of the present invention. the { at tem B 5ei stripped of encapsulation at 

FIG. 4 is a block diagram of sender and receiver circuitry eaeh j f 

for one station communicating using a LARQ (Limited Referring now t0 hg. 2, the lower three layers of the 

Automatic Repeat reQuest) protocol according to one network ^ are in ^ ^ fc well _ known> 

embodiment of the present mvention. $$ ^ iQ ^ seDder , s netwQrk layef w which forms 

FIG. 5 is frame map showing the fields of a conventional netW ork frames and passes the network frames to a data link 

Ethernet frame. layer 2 o, which in mm forms data frames from those 

FIG. 6 is a frame map showing the fields of a LARQ data ne twork frames and sends those data frames to a physical 

frame and its LARQ header. layer 30 It sbou ] d be understood that these layers are 

FIG. 7 is a frame map showing the fields of a LARQ 60 typically implemented as a combination of logic and storage 

reminder control frame and its LARQ header. that is configured to carry out the task of the layer. The logic 

FIG. 8 is a frame map showing the fields of a LARQ nack can either be in the form of hardware, software, firmware or 

control frame and its LARQ header. a combination of those. As is well-known, mass-produced 

FIG. 9 is an illustration of examples of some tables used and/or high-speed implementations are generally done in 
by. a LARQ handler, with FIG. 9(a) showing a channel state 65 dedicated hardware, while custom, special-purpose and/or 

information table, FIG. 9(b) showing a sender frame state low cost implementations might be done in software or 

table and FIG. 9(c) showing a receiver frame state table. firmware, with a general purpose processor executing 
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instructions to carry out the tasks of the layer. A layer might 
also be implemented, at least in part, using programmable 
gate arrays, or the like. More than one layer might be 
combined into an integrated circuit or software design. It 
should be apparent, upon review of this description, that 
there are many ways to implement the inventions described 
herein, using combinations of conventional circuit elements 
and/or software techniques. 

FIG. 3 illustrates how the networking model of FIG. 2 is 
modified to handle the LARQ protocol. As shown in FIG. 3, 
a LARQ handler 100 is included within the data link layer 
20. For data transmission, LARQ handler 100 accepts data 
frames from an upper interface 21 of the data link layer 20 
and outputs LARQ frames 102 to a lower interface 23 of link 
layer 20 (usually implemented within hardware under con- 
trol of a software driver). For data reception, LARQ handler 
100 accepts LARQ frames from the lower portion of the link 
layer and outputs data frames to upper interface 21. As 
explained below, some embodiments of LARQ handler 100 
can send LARQ and "non-LARQ" (i.e., conventional) 
frames as well as receive LARQ and "non-LARQ" frames, 
albeit with the non-LARQ frames not fully benefiting from 
the advantages of LARQ protocols. It should be understood 
from FIG. 3 that the data link layer might include additional 
processing elements between LARQ handler 100 and either 
or both of the interfaces to adjacent layers. 

FIG. 4 is a block diagram of one circuit for implementing 
LARQ handler 100. Typically, the LARQ layer is a sub-layer 
within layer 2 (data link layer) in the ISO network structure. 
As illustrated in FIG. 4, LARQ handler 100 includes sender 
control logic 104 and receiver control logic 106. Sender 
control logic 104 handles the input of frames from the upper 
layer, the output of data frames to the lower layer and the 
input of control frames from the lower layer, as well as other 
control functions described herein. Sender control logic 104 
also controls a channel table 110 and a transmit buffer 114, 
which are described in more detail below. Receiver control 
logic 106 handles the inverse of those functions, such as the 
output of frames to the upper layer, the input of data frames 
from the lower layer and the output of control frames to the 
lower layer, as well as other functions described herein and 
also controls a reorder buffer 120 and a receive status table 
122, which are also described in more detail below. 

Only one LARQ handler 100 is shown in FIG. 4, but it 
should be understood that one LARQ handler 100 would be 
provided at each LARQ-capable node on the network. 
Furthermore, it should be understood from FIG. 4 that the 
receive side and the transmit side of one LARQ handler 100 
typically communicate with other transmit sides and receive 
sides more often than they communicate among themselves. 
It should also be understood that the functionality of the 
receive side and the transmit side might be less separated 
than is shown in FIG. 4. One reason for showing the 
separation of the two sides is to aid in the understanding of 
the operation of LARQ handler 100. In an actual 
implementation, design and processing efficiency might 
combine elements. For example, as explained below, chan- 
nel table 110 might contain information for both the transmit 
side and the receive side. 

In the preferred embodiment, a network using LARQ 
protocols operates according to a modification of the Eth- 
ernet protocol. This allows the use of the many standard 
tools and components that have already been developed and 
are easily available for use with the Ethernet protocol. In the 
Ethernet protocol, an Ethernet frame passed between a data 
link layer and a physical layer has the structure shown in 
FIG. 5. 
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The frame shown in FIG. 5 is a standard Ethernet/IEEE- 
802.3 MAC (Media Access Control) frame and header with 
48-bit addressing. The frame includes a six byte (48-bit) 
destination MAC address, a six byte source MAC address, 

5 a two byte type/length field, followed by a pay load and a 
four byte FCS (Frame Check Sequence). The pay load is the 
data being carried by the frame and can be from 0 bytes to 
1500 bytes long. 
FIG. 6 shows the structure of a LARQ data frame 102, as 

10 might be passed between a LARQ layer and a physical layer. 
As shown there, LARQ data frame 102 is a modified 
Ethernet frame and comprises destination and source 
address fields, a LARQ header field, a type/length field, a 
payload field (that contains the data being transmitted) and 

15 possibly an FCS field. Atypical LARQ header comprises the 
fields shown in Table 1. Note that not all of the fields in Table 
1 are required, so a LARQ header could be as small as, or 
smaller than, eight bytes or as large as fourteen bytes. In the 
various descriptions of LARQ frames, a minimum size of 

20 eight bytes is assumed for a LARQ header. 

TABLE 1 



LARQ HEADER FIELDS 



Label Name (bits) Uses/Description 





Ethertype 


Ethertype 


16 


Assigned to Epigram by the IEEE 




Subtype 


Subtype 


8 


Assigned to LARQ by Epigram 


30 








(sample value: 0x10) 


Len 


Length 


8 


Additional length of LARQ header 








(either 4 or 10 bytes for a total 
length of 8 or 14 bytes) 




LARQ 


LARQ Version 


8 


Indicates the version of LARQ 




Ver. 






used by the frame originator. 




Rsv 


Reserved 


1 


Reserved for future use. Currently 


35 








set to 0 by the sender and ignored 
by the receiver. 




Nack 


Nack 

Indication/Count 


3 


Nack-000:No Nack 
Nack-001: 

Nack one sequence number 
Nack-010-111: 


40 








Sequence of 2 to 7 Nacks 








If Nack is not 000, 
Ctl must be 1. 




Ctl 


Control Bit 


1 


Ctl=0 signals a frame that carries 
a standard Ethernet payload 
following the LARQ header. 


45 








Ctl=l signals a control- frame 








with no payload. 




Pri 


Frame Priority 


3 


Indicates the priority level of 
the frame. 




Rsv 


Reserved 


8 


Reserved for future use; makes for 
an 8 byte header 


50 


Seq# 


Sequence 


8 


Incremented by one, modulo 256, 


Number 




for every new data frame. 



A receiver can distinguish a LARQ frame from a non- 
LARQ frame by the Ethertype of the frame since the 

55 Ethertype of a non-LARQ frame would fall in the same 
place in a frame as in a LARQ frame. In the examples set out 
here, an Ethertype of 0x88 6c is used to designate LARQ 
frames. That 32-bit value has been assigned to Epigram, the 
assignee of the present application, by the IEEE, the entity 

60 that assigns Ethertypes. An Ethertype value can be distin- 
guished from length values because valid lengths must be 
less than 0x0600, while valid Ethertypes are greater than 
0x0600. Other protocols are supported for the single Ether- 
type by using different subtypes to signal the use of a 

65 different protocol. 

The FCS field is usually not present on the frames passed 
between the upper and lower interface portions of the data 
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link layer, as most implementations of the MAC layer are in 
hardware and handle the FCS directly, adding and stripping 
it transparently. With the FCS, the total size of the frame is 
1518 bytes. If there is an error in the received FCS, the frame 
will be dropped. If a LARQ handler is tightly embedded, it 5 
may have access to frames with bad FCS values before the 
frames are dropped, allowing for faster retransmission in 
some cases. 

When a frame is received with Nack set to other than 000, 
it indicates a request for retransmission. The nonzero value 1Q 
of Nack indicates how many frames are being "nacked" (up 
to seven). With this scheme, the Nack frame can only nack 
sequential frames. Thus, if Nack-001, then the Sequence 
Number field indicates the missed/lost frame, whereas if 
Nack-011, then the three frames starting with the indicated ^ 
sequence number are being nacked. For multiple, nonse- 
quential frames, multiple Nack frames would be sent, in the 
preferred embodiment. 

If there is bidirectional data flow, the nacks could be 
inserted into frames traveling from the "nacking" receiver to 20 
the sender, although the described implementation does not 
do so. If the data flow is unidirectional, the nacks would be 
sent in control frames (i.e., Ctl=l). If more than one frame 
in a sequence is lost, one nack can signal the loss of up to 
seven sequential frames, by setting the Nack field to the 25 
number of lost frames. 

If Nack-000 but Ctl-1, then the frame is a "reminder" 
frame from the sender to the receiver. In a reminder frame, 
the sequence number is set to the sequence number of the 
most recently sent data frame, in case receiver missed it. 30 
Otherwise, if the receiver completely lost that last sent 
frame, it might never nack that lost frame, since the typical 
event that alerts the receiver to the loss of a frame is the 
receipt of a frame from later in the sequence. 

The sequence field contains an eight bit sequence number 35 
that is incremented by one, modulo 256, for each new data 
frame sent. It is not incremented for retransmitted frames, 
which are sent with the sequence field set to the frames' 
original sequence numbers. The sender maintains one 
sequence for each destination if the sender is sending frames 40 
to more than one destination. Upon sending a frame to a 
given destination, the sender only increments the sequence 
number for that destination, so that each destination receives 
sequential frames from each source sending frames to that 
destination. 45 

The destination can rely on receiving frames of a given 
priority level from a given sender in sequence, excepting, of 
course, frames that are lost. If multiple priority levels are not 
supported, then frames from a given sender are received in 
order, unless they are lost. Since receipt of a frame out of 50 
order indicates the loss of frames prior in the sequence 
relative to the received frame, the receiver would send nacks 
for the missed prior frames upon receipt of the received 
out-of-sequence frame. Since some frame handling systems 
allow for frame reordering in transit based on differing 55 
priorities of frames, the sender preferably uses a separate 
sequence number for each priority level. So that the receiver 
can determine the priority of a received frame (which is 
needed to determine which sequence to check against), a 
three bit priority field is included in the LARQ header, thus 60 
allowing up to eight different levels of priority. In other 
words, sequence numbers are assigned at a sender per 
destination and per priority level in order to prevent 
sequence number inversion problems in networks that allow 
frames with different priorities to be reordered at a lower 65 
layer, such as in the software driver or the hardware portion 
of the link layer, after the LARQ header has been added. 
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The maximum size of the payload field of LARQ data 
frame 102 might need to be reduced by eight bytes to adjust 
for the LARQ header if the underlying layers cannot handle 
a frame that is eight bytes longer. For example, in an 
intermediate driver within Microsoft's MS Win32 stacks, 
the Maximum Transmission Unit (MTU) for the upper layer 
could be specified to be shorter than normal. For networks 
where the physical layer (and its driver) will support frames 
at least eight bytes larger than the Ethernet maximum of 
1518 (1522 for VLAN), the MTU could be left unchanged. 

FIG. 7 shows the fields of a LARQ reminder control 
frame 102', which is similar to LARQ data frame 102, except 
that the typeAength field and the payload fields are not 
present. The Control field is set to 1 to signal that there is no 
payload field. If the underlying network requires a minimum 
frame size, the frame is padded to that minimum size. LARQ 
control frames are padded to Ethernet (or other physical 
layer) minimums as necessary, but otherwise are kept as 
short as possible to minimize network overhead. In Ethernet 
networks, the minimum frame size is 64 bytes, so the frame 
would be padded with a pad ranging from zero bytes to the 
minimum frame size. In some places in this disclosure, a 
LARQ header size of eight bytes is used by way of example, 
but other sizes such as seven bytes or six bytes might be used 
instead. 

FIG. 8 shows the fields of a LARQ nack control frame 
102". LARQ nack control frame 102" is similar to LARQ 
reminder control frame 102', except that LARQ nack control 
frame 102" includes a six byte nack extension field (and 
consequently, would need to be padded, if at all, by six less 
bytes). The nack extension field holds the destination MAC 
address of the original sender. This is the destination MAC 
address for the frame being nacked. For unicast destinations, 
this address is the same as the source address of the control 
frame. For group destinations (multicast/broadcast), this will 
be the original group address to which the frame was sent. 
The extra address is required because, when a station sends 
a frame, it must use its own MAC address as the source 
address for the frame, as it cannot use a multicast address for 
protocol reasons. When a multicast frame is received, the 
frame's destination address is the multicast address, not the 
station's own MAC address. The original sender needs to 
know the multicast address that was used for the frame being 
nacked, and doesn't care about the receiver's MAC address. 
Further, the LARQ nack must be sent back to the original 
sender, since there may be multiple senders to the same 
multicast address, each with its own sequence number space. 
This is because each combination of source, destination and 
priority is a separate logical channel and each logical 
channel has its own sequence number space. 

Referring now to FIG. 9, tables maintained by the stations 
are shown in that figure. Each sender maintains a channel 
table, such as the channel table 110 shown in FIG. 4 and in 
further detail in FIG. 9(a). Each entry in channel table 110 
represents one channel between the sender and a receiver. 
Other data maintained by senders and receivers are shown in 
FIGS. 9(b)-(c). If a sender is sending different priority 
frames to a given receiver, separate logical channels are 
used. 

Using the LARQ frames described above, both LARQ 
and non-LARQ frames can coexist on an Ethernet network. 
Each sender notes, in its channel table, whether or not 
destinations are LARQ-capable, Conventional Ethernet 
frames are sent unless both the sender and the receiver are 
LARQ-capable. It is possible to have the entire network be 
LARQ capable, but that is not necessary. Apacket processor 
can easily distinguish between a LARQ frame and a non- 
LARQ frame, as explained above. 
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Tlie handling of nacks will now be described wilh refer- numbers are missed (S17). Alimit on frame storage might be 

ence to the flowcharts of the transmission process for LARQ reached because the sequence number of the current frame 

frames (FIG. 10) and the reception process for LARQ might cause the sequence numbers for frames in the reorder 

frames (FIG. 11), as well as LARQ handler 100 shown in buffer to go out of range. Rather than continue to wait for the 

FIG. 4. A basic implementation will be described first, 5 missing frames missing between the Last sent-up frame and 

followed by examples of variations applicable to more the oldest frame in the reorder buffer, those missing frames 

specific embodiments. The logic of FIGS. 10 and 11 might are declared lost and the oldest frame is sent up. All frames 

be performed by sender control logic 104 and receiver in sequence after that oldest frame up to the next gap in 

control logic 106, respectively, as shown in FIG. 4. sequence numbers are sent up at this point. 

The transmission process is shown in FIG. 10. The 30 If the next higher layer does not require frames to be 

flowchart shown there omits some peripheral steps, for delivered in order, the LARQ handler will pass up frames as 

clarity of what is shown. For example, the flowchart does not they are received, rather than storing the out of order frames, 

address setting up new channels and operates on frames However, where the next higher layer requires frames in 

from existing logical channels. It should be apparent from order, or assumes the loss of frames if they are out of order, 

the remainder of this disclosure how a receiver or sender sets 15 the LARQ handler should be configured to buffer frames 

up a logical channel and such setup can happen on the fly, following a gap for a time in a reorder buffer so that if the 

i.e., a receiver could set up a logical channel after, and in receiver can fill the gap with retransmitted frames in time, 

response to, receiving a frame for a logical channel not the frames can be passed to the next layer in sequence order, 

already set up at that receiver. It should also be understood [f t he sequence number of the current frame is not the next 

that many variations of what is shown in FIG. 10 are 20 m the sequence, it indicates that frames were missed. In 

possible without affecting the essential steps of the process. particular, the frames with sequence numbers between the 

For example, the process shown begins in a state where a t as t received sequence number and the current sequence 

LARQ handler is waiting to receive a frame from a lower aumber were missed. The LARQ handler sends nacks for 

layer and leaves that state when a frame is received. In some those missed frames (S18) and sets nack retransmit timers 

variations, not shown, the LARQ handler might leave that 25 f or those missed frames (S 19). As explained below in further 

state in response to a timer timeout and return to that state detail, if the gap is larger than some threshold, nacks are not 

after processing a timed event. sent for all frames in the gap, and the gap is left for the 

The initial state is labeled "SI" in FIG. 10 and subsequent higher layers to handle, 

steps from that state are labeled with state or step numbers A receiver sends a nack control frame when it detects a 

S2, S3, S4, etc., which are referenced here in the text in 30 missing sequence number from a sender, either due to 

parentheses. Once a frame is received from the lower layer, receipt of a later sequence number or because a payload 

it is checked (step S2) to determine if it is a LARQ frame. error prevented delivery of the rest of the payload to the next 

If it is not, the frame is sent up to the next layer (S3) and the higher layer. The nack contains the first sequence number of 

LARQ handler returns to the initial state SI. If the frame is ^ one or more missing frames and a count of the number of 

a LARQ frame, and the frame is a nack frame, it is processed missing frames. The nack is a request for retransmission of 

as a nack frame (S4) and the LARQ handler returns to the the indicated original frames. Upon receiving a nack, a 

initial state. At this point, if the frame is not a nack frame, sender should immediately retransmit as many of the indi- 

it is checked. cated frames as possible, in their original order. 

If the frame is bad (e.g., bad header or out of range 4Q At this point, the LARQ handler checks the frame to 

sequence number), it is dropped (S5) and the LARQ handler determine if it is a reminder control frame. If it is, the frame 

returns to the initial state. If the frame has a bad FCS but has is dropped (S20). If not, the frame is sent up if there are no 

a good header and the next sequence number for its logical other frames in the reorder buffer (S21), otherwise the 

channel, the sequence number is advanced (S6) and a nack current frame is stored in the reorder buffer (S22). The 

is sent for the current frame (S7). A nack retransmit timer is 45 LARQ handler then returns to the initial state, 

set for the nacked frame (S8), the current frame is dropped j n somc embodiments, if a "receive lost" timer expires 

(S9) and the LARQ handler returns to the initial state. If the (indicating the time for holding frames), the buffered frames 

header is damaged, but the sender and the sequence number are t0 tne Qext higher layer regardless of whether or not 

are identifiable, a nack might be sent. there are gaps in the sequence. Thus, the receiver will stop 

If the frame is a LARQ frame and it is a good frame, the 50 waiting for a frame, declare it lost, and send sequentially 
sequence number is checked. If the sequence number is not following frames up to the next layer. As a frame is sent to 
new and the sequence number is too old or duplicates a the next higher layer, the expected next sequence number for 
previously received sequence number, the frame is dropped the frame's sender and priority is incremented by one, 
(S10) and the LARQ handler returns to the initial state. modulo the size of the sequence number field. 
However, if the old sequence number is not a duplicate or 5S The receive process has now been described. The send 
too old, the receiver's tables are checked to determine if the process will now be described with reference to the flow- 
current frame is the oldest missing frame (Sll). If it is, the chart of FIG. 11. For the send process, the sender waits for 
current frame is sent up (S12) and all following frames an evcnt (S41), such as the receipt of a frame, the expiration 
stored in the reorder buffer are sent up as well (S13). If the 0 f a timer or receipt of a control frame (signaled from the 
current frame is not the oldest missing frame, it is stored in 60 rece ive side of the LARQ handler). When the sender 
the receiver's reorder buffer (S14). In either case, the LARQ receives a frame from the layer above, it accepts the frame 
handler then returns to the initial state. anc i a dds a LARQ header as described above (S42). The 

If the current frame is good and the sequence is new (has sender references its channel table to determine the next 

not yet been received for the logical channel), the sequence sequence number to use (S43). The sequence number for 

number for that channel is advanced (S15), old frames are 65 that channel (specific to a destination and a priority) in the 

sent up if a limit on frame storage is reached (SI 6) and the channel table is incremented, modulo the size of the 

LARQ handler checks to determine if any new sequence sequence number field (in the above case, modulo 256), 
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prior to the transmission of a new non-control frame. The starting sequence number, sets the "type" for that logical 

other fields of the LARQ header are initialized as described channel in the channel table to RECEIVER ("receiver does 

above. LARQ"), adds the LARQ header to the frame, and sends it 

The frame is then stored in a transmit buffer (S44) and the down to foe driver. If needed, a new row is added to the 

reminder timer for that frame is reset. Preferably, the trans- 5 channel table. 

mit buffer is sized to allow some number of frames to be If the scnde r does not know whether or not the receiver 

stored for possible quick retransmission. Preferably, the implements LARQ, the sender sets up the channel with a 

transmit buffer handles only a limited number of frames, to starting sequence number and a "may not do LARQ" type 

limit the amount of space required for the transmit buffer. As ( as is the case witn channel 5 in channel table 110 shown in 

explained below, a sender might be configured to allow for ™ FIG. 9). The sender then sends the data frame without an 

a maximum number of frames overall (MaxTxSaveCount), LARQ header, and also sends an LARQ reminder frame to 

a maximum number of frames for any given channel the receiver using the destination MAC address and priority 

(MaxTxSaveCountChannel) and a maximum time for hold- of the data frame. Should a nack be received from the 

ing the frame (MaxTxHoldlnterval). receiver, then the type for that channel is set to "receiver 

Tne LARQ data frame is then sent to the next lower layer 15 does ' If ™ iS T*^ f*i ^ nV ?™ °f 

for transmission to the receiver (S45). Since the LARQ rem^r attempts, the channel is marked receiver does not 

header begins with a standard 16 -bit Ethertype field assigned 0 7 " . . . tat^l a 

by IEEE, non-LARQ receivers will simply discard LARQ WheD rece , iv , er receives a frame J 1 * a " ^ helder 

J, c v . r ClU , , 4 . t on a new channel (new source, group destination address, or 

data frames as being of an Ethertype unknown to them. In . . . . > ...•■,•.,«.■# 4 - • 

n , nna ^ tinn t ADn ^,f, f^Zoc- ™ ™, Mn » , r , nAn 20 priority), the receiver initializes channel state information in 

normal operation, LARQ data frames are not sent to non- f • . . . L i c * L 1 if iL c 

TAr>^ ■ u**u j- j * cTAnr\A* f „ its receive status table entry for that channel. If the frame is 

LARQ receivers, but this discarding of LARQ data frames , * f - ' 4 -*u^. *u 

a a -ujui , a . 7 u *u a data frame, the receiver sets up to receive the frame as the 

is used, as described below, to detect whether a new receiver 1 c . K. . 1C c 

j j. jyf\ t next in-order frame for the channel. If the frame is a 

is LARQ -aware or not. . , c iU . ... . c . , 

reminder frame, then the receiver initializes as* if it had 

At this point, the sender returns to step S41 and awaits 25 missed ^ fTame ^ the sequence number ^ foe reminder, 

another event. Upon expiration of some timers, the sender and sends a nack back to the source ni& has the effect of 

takes an action (S46). For example, when the "sendhold" leUing the sending statioQ know that this receiver also 

timer expires, frames id the frame buffer older than a sup p 0rts larq for the channel. 

predetermined time are removed from the frame buffer. Receivers do not positively acknowledge frames received 

When a "remind" timer expires, a reminder frame is sent. A 3Q successmllVt Upon detecting a sufficiently large gap in the 

sending station sends a reminder after a period of inactivity sequence numbe rs from a particular sender, a receiver does 

during which no frames have been sent to a recently used nQt nack tfae frames in the gaj)j but simply rcsets its 

destination MAC address and priority. As explained above, number state for th at channel to the current sequence num- 

the reminder frame contains the sequence number of the ber and ]eaves the higher layers t0 deal with thc gap ^ 

sender's most recently transmitted data frame, allowing the 35 typical threshold for a sufficiently large gap, in one 

receiver(s) to detect lost frames relatively quickly that implementation, is a gap wherein the absolute value of the 

otherwise might not be missed until much later. Once the difference between the new sequence number and the pre- 

timer is dealt with, if no other timers have expired, the vk)U5 most recent sequencc number is &fiBXtT than the 

sender returns to step S41. maximum number of frames for which the receiver keeps 

If, at step S41, the sender receives a control frame, 40 state, 

signaled by the receive side if the receive and transmit sides Resource Limits 

of the LARQ handler are separate, the LARQ handler The sender controls how much data (in frames, bytes, or 

processes the control frame, first checking to determine if both) are saved for possible retransmission, and for how 

the frames is a nack frame. If the control frame is a nack i ongi This allows limits to be set on how late a frame may 

control frame (or a nack indication on a return data frame 45 bc retransmitted both in real time and how far out of order 

where piggybacking is used), the sender resends the nacked it might be when received. The LARQ protocol does not 

frames (S47), otherwise, if the control frame is not a nack require explicit channel initialization between stations. The 

control frame, the sender deals with the control frame as receiver also places limits on the time it„ill hold frames 

appropriate (S48). Either way, the sender then returns back waiting for retransmission of an earlier frame, and the 

to step SI to await another event. Before resending the 50 number of frames it will hold. The limits imposed by the 

nacked frame(s), the sender determines if the nacked frame sender and receiver need not be identical. If the sender is 

is available in the transmit buffer. If it is available, the sender willing to buffer more than the receiver, then the sender may 

resends the frame. Otherwise, the sender takes no action. waste some memory for frames that can never be retrans- 

If the nacked frame is not retransmitted by the sender, the mitted. If the receiver has larger limits, then it may send nack 
receiver will eventually stop sending nack requests and 55 requests for frames that are already unavailable, resulting in 
consider the frame lost. The receiver will then send any some extra control traffic on the network. The basic imple- 
succeeding frames stored in its reorder buffer up to the next mentation does not require that the sender buffering is 
higher layer. The error handling processes of those upper consistent with the receiver buffering, but some im piemen- 
levels will likely make a peer-to-peer request among those tations might include configuration handshaking, to reduce 
higher layers for retransmission of the lost frame. 60 wasted traffic. 
Startup and Reset In-Order Frame Delivery 

When an Ethernet data frame is passed to a LARQ handler Most modem protocols tolerate out-of-order delivery, 

for transmission on a new channel (new destination or except for those such as PPP designed exclusively for 

priority), the sender handles channel initialization in one of operation on point-to-point links. However, imperfect and 

two ways. If all stations on the network are known to 65 inefficient implementations of higher layer protocols, 

implement LARQ, or if another channel to the same receiver including both transport and application protocols, make 

is already using LARQ, then the sender simply selects a in-order delivery highly desirable. Further, out-of-order 
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delivery is impossible on a normal Ethernet (including 
bridged multi-segment networks). For many network types, 
including Ethernet-like networks, a LARQ implementation 
should preferably deliver successfully received frames in the 
order originally sent by the sender. While the reordering has 
been addressed before, some care is required to handle lost 
frames in a timely fashion if the lost frames are to be 
transparendy replaced with retransmitted frames. 
Size and Usage of Sequence Numbers: Old and New Num- 
bers 

The size of the sequence number field might be changed, 
but eight bits appears a good choice for operation over a 
range of network speeds. Twelve bits is another possibility. 
At speeds much higher than 10 Mbps, a few tens of frames 
might be lost during certain kinds of network disruption. It 
is desirable, for robustness, to have some slack in the 
sequence number space so that very long disruptions will be 
detected as large gaps (which in turn cause sequence number 
resets at the receiver) instead of sequence numbers wrapping 
around their modulo without detection. To avoid missing 
notice of the wrap around the modulo, the preferred embodi- 
ment targets a sequence number space that can be divided 
into new numbers, which follow the current number, old 
numbers which precede the current number and some 
numbers, designated "out-of- sequence" numbers, which 
precede the old number and follow the new numbers (which 
is possibly, since the number space loops back on itself). For 
example, with a number space of 256 numbers, if the current 
sequence number is C, numbers C-63 through C might be 
old numbers, C+l through C+64 might be new numbers (all 
modulo 256) and the rest of the numbers would be out-of- 
sequence numbers. 

"New" and "old" numbers are defined with respect to a 
current receive sequence number. For the sender, the current 
sequence is the one most recently assigned to a data frame 
received from the next higher layer for transmission. For the 
receiver, the current sequence number is the sequence num- 
ber of the most recently received new frame, or the sequence 
number of the frame first received when the channel was 
initialized. Note that the sender and the receiver might have 
different current numbers at some times. 

Formally, an old sequence number is one whose differ- 
ence from the current sequence number modulo the size of 
the sequence number space is less than or equal to zero, 
while a new sequence number is one whose difference from 
the current sequence number modulo the size of the 
sequence number space is greater than zero. "Recent" 
sequence numbers are old sequence numbers close enough 
to the current sequence number to have saved state 
information, possibly including a saved copy of a data 
frame. Beyond the recent sequence numbers are the out-of - 
sequence numbers, which may not be defined in all versions 
of the LARQ protocol, but when defined, are those very old 
or very new sequence numbers that should not be in use by 
either sender or receiver given the current sequence number. 

When a receiver receives a frame with an old sequence 
number, the frame will be processed as a normal receive 
frame as long as it does not duplicate a frame already 
received. A new frame is always processed as a normal 
receive frame and also results in nacks being sent if one or 
more previously new sequence numbers are now missing. If 
an out of sequence number is received, the receiver will 
simply treat it as a reset of the sequence number space. 

If the LARQ handler does not have to deliver frames to 
the next higher layer in order, then normal receive process- 
ing for a non -duplicate data frame, new or old, is to send the 
frame up to the next higher layer. For in-order delivery, an 



15 



20 



25 



30 



35 



50 



55 



65 



old frame is sent up the stack if it is the oldest outstanding 
missing frame, held if there are still older missing frames, or 
dropped if it is a duplicate. New frames are sent up the stack 
if there are no earlier, missing frames, or held if there are 
missing frames with earlier sequence numbers. In addition, 
advancement of the current receive sequence number moves 
the range of old sequence numbers, possibly causing old, 
held, frames to be sent up the stack as they become the oldest 
outstanding sequence number (e.g. current-63). 
Nack Handling and Aggregated Nacks 

In the basic implementation, a nack indication which is 
conveyed to the sender for each missed sequence number, on 
a per-channel basis. When a Nack control frame is received, 
the source and destination addresses in the Ethernet header 
must be reversed in order to perform the channel lookup. A 
nack control frame should not, by default, cause a new 
channel to be set up with the original sending station as a 
receiver. The nack is processed in the context of the sender. 
The sender normally retransmits all of the indicated frames 
that are still available, although the MinimumRtxInterval 
(see below for an explanation of the variables used here) 
may be applied to prevent too-frequent retransmissions of 
individual frames. 

As an optimization, the Nack control frame includes both 
a sequence number and a count of missing frames, so that a 
single Nack control frame may be sent for several, 
sequential, missing frames. When the number of missing 
frames exceeds the size of the Nack Count field (three, in the 
above examples), multiple Nack control frames are sent to 
request the entire range of missing frames. 
Extensions of the Basic Implementation 

The basic implementation sends a single reminder for the 
most recent sequence number after a period of inactivity. An 
extended implementation might allow for multiple remind- 
ers be sent, but there are several reasons why a single 
reminder, sent after a relatively long interval compared to 
nacks, appears to be the best choice. The reminder is sent 
only if there is no further traffic. If a second reminder were 
sent after a similar interval, the frame might be near the end 
of its "lifetime" before being dropped as old, making the 
second reminder even more likely to be useless. Also, the 
relatively long interval makes it much less likely that a 
single error event would corrupt both the original frame and 
the first reminder. The fact that there is no further traffic on 
the same channel suggests that this is not part of a stream of 
latency sensitive frames, so allowing a higher layer protocol 
to take up the slack in the case of a double loss seems a good 
tradeoff, compared with almost doubling the number of 
overhead frames on the network. 

Nacks, on the other hand, should be resent if there is a 
good reason to believe that the first nack was unsuccessful. 
The procedures below are intended to cover this case, ideally 
through the use of adaptive timeouts to maximize the 
number of retransmission opportunities. 

A basic implementation might not have multiple 
priorities, in which case, the priority field can be left off of 
the LARQ header, but the priority field should be used where 
the lower layers do reorder frames based on a priority. 
Group Addresses (Multicast/Broadcast) 

The basic implementation can be extended for use with 
group addresses by defining channel state to be kept for each 
combination of source, destination and priority, and by 
extending the nack control frame format. In this extension, 
the nack control frame includes the original destination 
MAC address of the frame, since the receiver's own MAC 
address, used as the source address in the nack control 
frame, cannot be a group address. 
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Interval DescriptioD 



Piggyback Nacks, Fragmentation, Selective Operation, etc. 

An extended format header, with two sequence number 
fields, can be used so that every frame could carry piggy- 
backed nack indications. This can minimize the number of 
extra control frames generated by a LARQ handler for 5 
bidirectional traffic flows, but with a cost of extra header 
space for every frame, additional complexity, and some 
additional latency if nacks are held some brief interval while 
awaiting a potential data frame on which to get piggybacked. 
Other Extensions 10 

Several other extensions of the basic protocol and handler 
are contemplated. For networks on which longer frames are 
significantly more likely to be damaged or lost, it is possible 
to fragment larger frames ioto two or more segments. Since 
non-LARQ frames are not normally processed by a LARQ 15 
receiver expecting a LARQ frame, it is possible to add 
LARQ headers only to certain types of frames (perhaps 
based on protocol or priority). It is also possible to enable 
LARQ operation only when the error characteristics of the 
network become worse than some specified threshold. 20 

The basic LARQ protocol can be easily modified, using 
the teachings herein, for use over almost any type of network 
that provides a mechanism for defining logical channels if 
the network supports more than two stations. 
Parameter Negotiation 

Another possibly useful extension to LARQ is the accom- 
modation of parameter negotiations, or at least configuration 
announcements. For instance, the reminder and nack frame 
exchange used to probe and acknowledge the use of LARQ 
on a new channel can be exploited. The reminder frames can 
be extended to optionally contain the values of the sender's 
configuration, and nack frames could optionally contain 
values for the receiver's configuration. The sender would 
provide values for the logical channel for 
MaxTxSaveCountChannel, MaxTxHoldlnterval, and 
MinimumRTXInterval, while the receiver would send 
MaxRxSaveCountChannel, MaxRxHoldlnterval, and Repe- 
atRequestlnterval (see below for an explanation of these 
configuration variables). 

A receiver could use these variables to adapt its own 
per-channel configuration, especially when the sender's 
resource limits were lower than the receiver's default limits. 
Also, the MiniinurnRtxInterval might help prevent wasted 
nack retransmissions. A sender would also be able to limit 
the number of saved frames or the time that saved frames 45 
could be held to smaller values if the receiver had lower 
limits. 

A Detailed Example 

A more detailed specification for a specific embodiment 
of a LARQ handler in a network supporting the LARQ 50 
protocol will now be described. 

In this specific embodiment, the LARQ protocol is first 
defined for a logical channel between a pair of stations with MaxTxSaveCount 
user data in a single direction. The basic logical channel is 
defined by the tuple comprising the source and destination 55 
MAC addresses of the user data frames. An extended 
definition of the logical channel adds the priority of the user 
data to the tuple. Although not part of standard Ethernet, 
priority is available as an extra parameter for frames in 
Win32 network stacks, and is supported for some network 60 
types that use IEEE Ethernet headers, including a phoneline- 
based network developed by Epigram. 

A frame not yet received is declared "missing'' if its 
sequence number is between the most recent known 
sequence number and the oldest possible sequence number 65 
for which the receiver would accept a retransmission. A 
frame is declared "lost" if it is missing when the receiver 



decides it must send up a frame with a more recent sequence 
number, whether due to timing, 'resource, or sequence num- 
bering limits. If a frame is received after it has been declared 
lost, it should be discarded. 

This specific implementation keeps track of time in units 
of milliseconds, using a 32-bit millisecond tick counter 
maintained by the system for time-stamping various events. 
Timers are implemented by computing the interval (in 
milliseconds) to the next timer-based event for each logical 
channel, and calling a WaitForNextEvent function. 

TABLE 2 



TIME-OUT INTERVAL TIMERS 



25 



30 



35 



The maximum time that transmitted frame [or copy] will be 
held by a sending station's LARQ handler waiting for 
possible retransmission. 

The maximum time that a receiver will wait, after 
detecting a Lost (missing or errorcd) frame, before declaring 
the frame to be lost. When a frame is declared lost, 
succeeding frames that were being held awaiting 
reception of the lost frame can then be sent up the 
stack to the next higher layer. By default, 
this parameter has the same value as T (endHcld . 
The duration of a period of inactivity after the last data 
frame transmitted on a channel that must expire before 
sending a reminder control frame. 
The minimum time interval between successive 
retransmissions of the same data frame. If a second request 
for the same frame is received before this interval has 
expired, the request will be ignored. This is a function 
of the smoothed average retransmission request interval 
seen by the sender for the channel. 
The interval between successive retransmissions of 
a nack for a missing frame. This is a function of the 
smoothed average retransmission interval seen by the 
receiver for the channel. 



The LARQ handler also maintains a set of configuration 
parameters that control the operation of the LARQ protocol 
with respect to that LARQ handler. Some of these param- 
eters are shown in Table 3 along with a description and 
typical value for each configuration parameter. 

TABLE 3 

CONFIGURATION PARAMETERS 



Parameter 



Description 



Typical 
Value 



MaxTxSaveCountChannel 



MaxRxSaveCountChannel 



MaxRxSavcCount 



MaxProbeCount 



The maximum number of 
frames that will be held by a 
sender for a single channel. 
The maximum number of frames 
that will be held by a sender 
for all channels. 

The maximum number of frames 
that will be held by a receiver 
for a single channel to wait for 
a missing, but not yet declared 
lost, frame to arrive. 
The maximum number of frames 
that will be held by a receiver 
for all channels. 

The number of reminder probes 
to be sent before deciding that 
a receiver does not do LARQ 
on a particular channel. 



30 



depends 

on 
available 
memory 

30 



depends 

on 
available 
memory 

10 
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TABLE 3-continued 



TABLE 4 



CONFIGURATION PARAMETERS 



typical 



Parameter 


Description 


Value 


MaxTxHoldlnterval 


The initial value foi T, endHo4(L 


150 ms * 


MaxRiHoldlnterval 


The initial value for T IcvLo<1 . 
This is the interval it takes to 
declare a frame as lost and is 
also the longest interval that a 
more recent frame can be held 
befoTe it is sent up the stack. 


150 ms 


Remindeilnterval 


The initial value for T Icajind . 


50 ms 


MinimumRtxInterval 


The initial value for T mtnRrx 


10 ms 


Repea LReques tin terval 


The initial value for T rcquc „. 


25 ms 



A "slow" device with a relatively low maximum frame 
rate might use a small value for MaxTxSaveCountChaiinel, 
such as one or two, while high-speed devices will need much 
greater values. With a sequence space of 256 sequence 
numbers and using the above scheme of old, new and invalid 
sequence numbers, the upper limit for MaxTxSave- 
CountChannel is 64, since that is one fourth of the sequence 
number space. 

MaxTxSaveCount is preferably a function of 
MaxTxSaveCountChannel, the number of channels, the 
maximum data rate of the network, the amount of system 
memory, etc. 

MaxRxSaveCountChannel should be less than or equal to 
the value of MaxTxSaveCountChannel, and it is possible to 
use one parameter for control both functions. If a receiver 
already has MaxRxSaveCountChannel frames stored in its 
reorder buffer, this limit would be exceeded when a new 
frame is received. Therefore, if that condition exists, the 
receiver should send the oldest saved frame up to the next 
layer and have the missing frames older than the sent up 
frame declared lost. 

MaxProbeCount is the number of reminder probes to be 
sent before deciding that a receiver does not do LARQ on a 
particular channel. If the traffic to a receiver is such that 
MaxProbeCount frames are sent in a quick burst, the sender 
may quickly conclude that the receiver is not LARQ- 
enabled, even if it is. To avoid this situation, it may be 
desirable to add a timeout interval so that only one reminder 
probe is sent out in each timeout interval. 

The following Tables 4-8 set out some informa tion that is 
maintained for the LARQ processes. Table 4 sets out the 
information maintained for each channel known to a LARQ 
layer handler, whether the LARQ layer handler is using the 
channel as a receiver or a sender. As explained above, each 
sender-destination-priority permutation has its own logical 
channel. An example of the data for such a channel state 
information table is shown in FIG. 9(a). Some of the fields 
shown in Table 4 are omitted from FIG. 9(a) to keep the 
illustration a reasonable size. 

Table 5 sets out the information maintained for each 
channel at the sender end of the channel. Table 6 sets out the 
information maintained for each frame stored by the sending 
LARQ layer handler. An example of sender data is shown in 
FIG, 9(b). Table 7 sets out the information maintained for 
each channel at the receiver end of the channel. Table 8 sets 
out the state information stored at the receiver for each frame 
being handled by the LARQ handler. An example of the data 
for receiver data is shown in FIG. 9(c). 



Field 



CHANNEL STATE INFORMATION 



Description 



Channel Identification 
SenderlD 
Destination! D 



15 



Priority 
Channel Type 



20 



25 



Cur Seq 
35 Oldest Seq 

Frame Table 
Probe Count 



40 



A channel number. 

The MAC address of the sending station. 
The destination MAC address used by the 
sending station. 

This is usually the receiving station's 
MAC address for unicast frames and the 
destination group MAC address for 
multicast frames. 

The priority of the frame as specified 
by the sender. 
A type selected from: 
UNKNOWNRECEIVER, NO RECEIVER, 
NOSENDER, RECEIVER, SENDER 
UNKNOWNRECEIVER indicates that the 
sender does not know if the receiver 
supports LARQ. Up to MaxProbeCount 
reminder control frames are sent as 
probes, one with each frame sent on 
the channel. If a positive response 
is received the type is changed to 
RECEIVER, otherwise the type is changed 
to NORECEIVER 

NORECEIVER indicates that the receiver on 
this channel is not supporting LARQ. 
NOSENDER indicates that the sender on this 
channel is not supporting LARQ. Receipt of 
an LARQ frame causes the type to change to 
SENDER. 

RECEIVER indicates that this station is 
operating as the receiver for the channel. 
SENDER indicates that this station is 
operating as the sender for the channel. 
The current sequence number for this channel. 
The oldest sequence number processable for 
this channel. 

A reference to a frame table. 
The number of unsuccessful probes sent to the 
receiver on a channel in the 
UNKNOWNRECEIVER type. 



TABLE 5 



45 



Field 



SENDER CHANNEL STATE INFORMATION 
Description 



Send Sequence Number 



50 



Retransmission Interval 



55 



60 



SendFrameState 
[MaxTxSaveCount] 



This eight bit field is incremented by one, 
modulo 256, for each new user data frame to 
be transmitted to the receiver. 
The new value is placed in an LARQ header 
that is inserted into the frame before sending 
it on to the driver. 

As implemented, this is a function of a 
smoothed mean and smoothed mean deviation 
for retransmission intervals measured from the 
time a frame was sent to the time it was 
retransmitted, using dynamic tracking of 
frame round-trip travel lines. For some 
implementations this could be a fixed or 
configurable constant value. The current 
implementation uses a starting value of 
25 milliseconds. 

This is an array of size MaxTxSaveCount. 
State information is kept for each user frame 
for as many as MaxTxSaveCount sequence 
numbers. 

The state information may include, for each 
frame, a copy of the frame as well as the fields 
shown in Table 6. 
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TABLE 6 



Field 



SENDER FRAME STATE INFORMATION 



Description 



TxFramcFlag 1 = have saved frame, 0 ** no saved frame. 

This field could be an implicit field, where only 
saved frames are listed in the table. 
Retransmission Flag 1 = retransmitted at least once, 0 = not yet resent. 

This field might not be explicit; it could be 
implicitly determined from Retransmit Time. 
If TxFrameFlag is 1, this field is a copy of the 
frame that can be used for retransmission. 
This may be a pointer to frame data. 
The time at which the frame was originally sent. 
The time when the frame was retransmitted, valid 
only if Retransmission Flag is 1. 



Send Frame 



Send Time 
Retransmit Time 



TABLE 7 



RECEIVER CHANNEL STATE INFORMATION 



Field 



Description 



Receive Sequence Number 
Mean Request Interval 



ReceiveFrameTable 
[MaxRxSaveCnt] 



The most recent sequence number received 
on the logical channel. 
As implemented, this is a function of a 
smoothed mean and smoothed mean 
deviation for retransmission intervals 
measured from the time the frame was 
missed to the time the retransmitted frame 
was received. For some implementations, 
this could be a fixed or configurable 
constant value. A typical starting value 
is 25 milliseconds. 

This is an array of size MaxRxSaveCnt. 
State information is kept for each user 
frame for as many as MaxRxSaveCnt 
sequence numbers. The state information 
may include, for each frame, a copy of the 
frame as well as the fields shown in Table 8. 



TABLE 8 



Field 



RECEIVER FRAME STATE INFORMATION 



Description 



20 



Received Flag 1 = a good frame was received, 0 = frame not received 
(either at all, or it was errored). This field might be 
implicit, if only good frames are stored in the table. 

RxFrame Flag 1 - frame is being held, 0 - no frame being held. 

Rx Lost Flag 1 - sequence number was timed out and higher 

numbered frames are no longer held for it, 0 - frame 
not declared "lost". 

Receive Frame If RxFrame Flag is 1, this field is the frame that is being 
held (or a pointer to frame data), waiting to be sent up to 
the next layer. 

Miss Time The time the frame was first missed (i.e., when a higher 
numbered frame was received, or an errored copy of the 
frame was received). This field is used for computing the 
average retransmission interval from the receiver's 
perspective. 

Nack Req Time The time that the last nack request was generated. Used 

for generating the next retransmission request timeout 
Receive Time The time that the frame was received (time stamp). 



The first three or four fields of receiver frame state 
information could be explicit or could be implicitly deter- 
mined from the time stamp (Receive Time). An example of 
a communication session using the above tables and param- 
eters will now be described. 
Channel Initialization 

To begin communications between one sender and one 
receiver, a logical channel is initialized. The only explicit 



control communication needed between senders and receiv- 
ers can be handled using nack indications. When a sender 
first uses a new destination/priority, the sender's LARQ 
module sets up a new logical channel and assigns an initial 

S sequence number for the first frame. All timers are initially 
disabled, the list of saved frames for the new channel is 
cleared, channel parameters are initialized to their default 
values, and other state information (counters, etc.) is appro- 
priately initialized. 

10 If the receiver is known to implement LARQ, then the 
type of the channel is set to SENDER. Otherwise, the 
channel type is set to UNKNOWNRECEIVER. At this 
point, the sender sends a reminder control frame to the 
receiver and the channel variable ProbeCount (initialized to 

15 0) is incremented (to 1). 

When a receiver first receives a frame with an LARQ 
header for a new channel, where a new channel is any 
communication for a new source, from a new destination or 
with a new priority, the receiver sets up a new logical 

20 channel. The list of held frames for that new channel is 
cleared, if needed, as are all counters and similar state 
variables for that channel, and channel configuration param- 
eters are set to their initial values. The state is initialized to 
accept the new frame as the next in-sequence frame by 

25 setting the Receive Sequence Number to the sequence 
number preceding that in the new frame, and marking all 
sequence numbers for which state is maintained as received. 

A receiver may find a new sequence number from a source 
to be way out of sequence, either due to many lost frames or 

30 a reset by the sender. In this case, the receiver simply sends 
up all saved frames and resets its channel state as if the new 
sequence number were the first from the source on the 
logical channel. However, statistics counters for the channel 
are not reset. 

35 Following the probe, the sender begins sending data 
frames. 

Sending a Data Frame 

In an Ethernet network, LARQ is implemented as a 
protocol layer between the Ethernet link layer and the 
40 Ethernet network driver. Thus, Ethernet frames that are 
ready for transmission onto the network pass through the 
LARQ protocol layer. When a LARQ transmit module 
receives an Ethernet frame for transmission, it checks its 
channel table to determine if the frame is destined for a 
45 logical channel that already exists and is of type SENDER. 
Otherwise, a new channel is initialized as described above. 

If the channel already exists, the Send Sequence Number 
for the channel is incremented, and its new value is included 
in an LARQ header inserted into the frame immediately 
50 following the source MAC address field. The Priority field 
is set to the priority specified by the next higher layer, or zero 
if the sender does not use priorities. The Control Flag is set 
to zero, as are other unused fields in the LARQ header. The 
frame is then passed down to the driver for normal trans- 
55 mission. The time that the frame was first transmitted is 
recorded. A logical timer is started, set for the interval 
T sendJtold , whose expiration will cause the save frame to be 
released, unless the timer was halted due to early release of 
the frame for other reasons. A logical reminder timer, 
60 described in the next section, is also (re)started. 

If the sender's channel type is NORECEIVER for the 
channel matching the new to be sent, the frame is sent to the 
lower layer driver unchanged (without an LARQ header). 
When the type is UNKNOWNRECEIVER, the new data 
65 frame is sent unchanged, but the sender also checks the 
Probe Count parameter for the channel. If Probe Count is 
less than MaxProbeCount, the sender sends a reminder 
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control frame with the channel's destination and priority the current Receive Sequence Number), it is dropped with 

(without incrementing the initial value of the Send Sequence suitable statistics gathering. If the received frame's sequence 

Number), and increments Probe Count. If Probe Count has number is recent (within MaxRxSaveCount of Receive 

reached its limit, the channel's type is changed to NORE- Sequence Number) but has already been marked as received 

CEIVER. 5 or l° st » ^ is dropped with suitable statistics. If not dropped, 

Reminder Control Frame Generation a received frame with a recent sequence number will be 

In order to protect individual frames or those that are the marked as received. The receiver may compute statistics on 

last in a series, a short reminder control frame is sent after the d c me taken t° receive a retransmitted frame from when it 

an interval has passed with no further frames to be trans- w / s *f missed m order to su PP ort ada P tlve retransmission 

. or nicies 

mitted. The reminder control frame contains the sequence 10 ™ ' >1P , . . , t . - .« u 

. c tU . . •** j & c j Then, if reordering is not implemented, the frame will be 

number of the most recently transmitted frame Send ^ (0 ^ next hi | faer { ^frames m being reordered 

Sequence Number, so that a receiver can detect the frame s and the received frame does not have me oMest missing 

loss and send a nack. The logical implementation for this sequence number, the frame is saved, and the RxFrame Flag 

function uses a timer that is set to T nmind and is started each for its sequeace numbcr is set to 1 . If the received frame has 

time a new frame is transmitted on a channel. Expiration of is tne oldest missing sequence number, the frame is sent up to 

the timer causes a reminder control frame to be transmitted. tne nex t higher layer, along with any further, in-sequence, 

The timer can either be restarted after expiration of the timer, saved frames. As saved frames are sent up, the RxFrame 

or in some embodiments, the timer is not restarted until the Flag for each is set to 0. 

next frame is sent. If a received frame's sequence number (not a Nack 

Handling Nacks and Retransmissions 20 control frame) is new and within a window of MaxRxSave- 

When a nack control frame is received, the source and CountChannel from Receive Sequence Number, the receiver 

destination addresses in the Ethernet header are logically will update its state by advancing the window of recent 

reversed in order to perform the channel lookup. The nack sequence numbers until the received frame's sequence num- 

is processed in the context of the sender, A nack control ber is current. If the received frame's new sequence number 

frame should not be used to cause a new channel to be set 25 was outside of the valid sequence numbers, the sequence 

up with the original sending station as a receiver, because the number should have been treated as out-of-sequence, and the 

only sequence number in the LARQ header applies to the channel reset function performed so that the new frame will 

original sender's channel. The sender normally retransmits be in-sequence. 

all of the indicated frames that are still available, al though The Receive Sequence Number is repeatedly incremented 

the MinimumRtxInterval may be applied to prevent too- 30 by 1 (modulo 256, or other size of the sequence space) until 

frequent retransmissions of individual frames. it is equal to the received frame's sequence number. Each 

When a nack has been receives each of the indicated time it is updated, the state of the new current sequence 

sequence numbers is processed, from oldest to newest. If the number is initialized as missing and the time when it was 

sequence number is recent and the frame is still available, first missed is recorded, unless the current number is that of 

the sender checks whether the frame has been retransmitted 35 the receive frame and the receive frame was a valid data 

at all Retransmission Flag). If not, the frame is immediately frame (not a reminder and not errored). If the frame is 

retransmitted, the Retransmission Flag is set, and a times- marked received, it is also saved, possibly temporarily. For 

tamp of the retransmission time for the frame is stored, each new sequence number, the trailing edge of the sliding 

Some implementations may update statistics on the interval window of recent sequence numbers also changes. The new 

between the original sending of the frames and retransmis- 40 oldest recent sequence number is checked to see there is a 

sioos. If the frame had previously been retransmitted, then held frame. If there is a saved frame (Rx Frame Flag-1), that 

the interval from its previous retransmission to the present is frame is sent up to next higher layer and Rx Frame Flag is 

computed and compared with Retransmission Interval. If too set to 0. When the current sequence number has been fully 

little time has elapsed, then the frame is not retransmitted, updated to the received sequence number, the receiver then 

otherwise the frame is retransmitted and its retransmission 45 scans the history of recent frames, starting with the oldest 

timestamp is updated. sequence number not yet lost or sent up. If that sequence 

Reception of Frames number has a held frame, then that frame and any 

Receive d frame s with the LARQ Ethertype are pro- in-sequence held frames that folio w it are sent up to the next 

cessed by the LARQ protocol module. Nack control frames higher layer. This will result in the just-received frame to be 

are handled independently from other received LARQ 50 sent up to the next higher layer, if appropriate, 

frames, as described above. LARQ data frames and Lost Frame Timeouts 

reminder control frames are handled in a similar manner to At the time a missed sequence number was first noted, a 

each other since most of the complex processing involves timestamp was recorded. Logically, a timer with value 

management of sequence number state. When the LARQ T n . v7 ost was started for that sequence number. If the frame is 

module has determined that a frame should be sent up to the 55 received then the timer is (logically) stopped. If the frame is 

next higher layer, the LARQ header is removed so that the never received and the timer expires, then the sequence 

Ethernet source and destination MAC addresses are once number is marked lost (Rx Lost Flagyl). When a sequence 

again contiguous with the original Ethernet Type/Length number is marked lost, the next higher sequence number is 

field, and the frame is passed up. Frame handling depends checked to see if a held frame is present. If so, the held frame 

both on the sequence number of the received frame and on 60 (now the oldest, not yet sent up, frame) is sent up to the next 

the state of recent sequence numbers. Some implementa- higher layer, along with any in-sequence held frames that 

tions may choose not to implement in-sequence delivery of follow it. 

frames to the next higher layer, in which case the handling The above-described processes repeat as long as the 

of the actual receive frames is much simpler, since no channel remains open. FIGS. 12^13 show examples of how 

buffering is required, 65 LARQ frames might be handled. 

If the received frame's sequence number is not out-of- FIG. 14 shows a typical network 200 where the present 

sequence but not recent (older than MaxRxSaveCount from invention might be used. As shown in FIG. 14, network 200 
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uses existing residential telephone wiring to form a network. 
A personal computer 202 contains a network card 204 that 
connects to network 200 via a connection to an RJ-11 jack 
206, Other network devices, such as a networked appliance 
208 (a printer, VCR, environmental controller, or the like), s 
might also be coupled to network 200 in a similar manner. 
Using the LARQ protocol, the devices connected to network 
200 could communicate at higher effective data rates than if 
LARQ were not used. 

The above description is illustrative and not restrictive. 10 
Many variations of the invention will become apparent to 
those of skill in the art upon review of this disclosure. 
Merely by way of example, the above description described 
modifications to an Ethernet network to support the LARQ 
protocol, but the LARQ protocol is not limited to such a 15 
network and can be easily modified, using the teachings of 
this disclosure, to operate over other network types. The 
scope of the invention should, therefore, be determined not 
with reference to the above description, but instead should 
be determined with reference to the appended claims along 20 
with their full scope of equivalents. 

What is claimed is: 

1. In a frame-switched network apparatus, a method of 
sending frames from a sender to a receiver over a possibly 
unreliable channel, the method comprising the steps of: 25 

forming a frame at the sender, wherein the frame contains 

data to be transmitted to the receiver; 
including a frame identifier in the frame selected from a 

set of frame identifiers; 

30 

retaining a copy of the frame at the sender; 

sending the frame from the sender to the receiver over the 

channel, independent of a presence of the receiver on 

the possibly unreliable channel; 
upon receipt of a frame at the receiver, identifying a frame 35 

identifier for the received frame; 
detecting, from the frame identifier, if a prior frame was 

missed; 

if a missed prior frame is detected in the step of detecting, 
sending a negative acknowledgment (nack) at least two 40 
times from the receiver to the sender, the nack includ- 
ing an indication of the missed prior frame; 

if a nack is received at the sender, determining the frame 
identifier of the missed prior frame and resending the 45 
missed prior frame if a copy of the missed prior frame 
is still retained at the sender; 

delaying a second nack from the receiver for a response 
period, wherein the response period is related to the 
time delay expected between sending the first nack and 50 
expected receipt of a retransmitted frame and is a 
dynamically determined time determined from mea- 
sured frame travel times;. 

retransmitting the missed prior frame once for each nack 
received; and 55 

releasing, independent of an acknowledgment of a suc- 
cessful receipt of the transmitted frame by the receiver, 
the retained copy of the transmitted frame when a 
storage constraint is reached. 

2. In a frame-switched network apparatus, a method of 60 
sending frames from a sender to a receiver over a possibly 
unreliable channel, the method comprising the steps of: 
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24 

forming a frame at the sender, wherein the frame contains 

data to be transmitted to the receiver; 
including a frame identifier in the frame selected from a 

set of frame identifiers; 
retaining a copy of the frame at the sender; 
sending the frame from the sender to the receiver over the 

channel, independent of a presence of the receiver on 

the possibly unreliable channel; 
upon receipt of a frame at the receiver, identifying a frame 

identifier for the received frame; 
detecting, from the frame identifier, if a prior frame was 

missed; 

if a missed prior frame is detected in the step of detecting, 
sending a negative acknowledgment (nack) from the 
receiver to the sender, the nack including an indication 
of the missed prior frame the indication of the missed 
prior frame including a frame identifier of a first missed 
frame and a number of sequential missed frames fol- 
lowing the first missed frame; 

if a nack is received at the sender, determining the frame 
identifier of the missed prior frame and resending the 
missed prior frame if a copy of the missed prior frame 
is still retained at the sender; and 

releasing, independent of an acknowledgment of a suc- 
cessful receipt of the transmitted frame by the receiver, 
the retained copy of the transmitted frame when a 
storage constraint is reached. 

3. In a frame -switched network apparatus, a method, of 
sending frames from a sender to a receiver over a possibly 
unreliable channel, the method comprising the steps of: 

forming a frame at the sender, wherein the frame contains 
data to be transmitted to the receiver; 

including a frame identifier in the frame selected from a 
set of frame identifiers; 

retaining a copy of the frame at the sender for a buffer 
period and tracking the buffer period for each frame; 

sending the frame from the sender to the receiver over the 
channel, independent of a presence of the receiver on 
the possibly unreliable channel; 

upon receipt of a frame at the receiver, identifying a frame 
identifier for the received frame; 

detecting, from the frame identifier, if a prior frame was 
missed; 

if a missed prior frame is detected in the step of detecting, 
sending a negative acknowledgment (nack) from the 
receiver to the sender, the nack including an indication 
of the missed prior frame; 

if a nack is received at the sender, determining the frame 
identifier of the missed prior frame and resending the 
missed prior frame if a copy of the missed prior frame 
is still retained at the sender; and 

releasing, independent of an acknowledgment of a suc- 
cessful receipt of the transmitted frame by the receiver, 
the retained copy of the transmitted frame when a 
storage constraint is reached. 

***** 
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